Efficient and effective ER with progressive blocking

Blocking is a mechanism to improve the efficiency of entity resolution (ER) which aims quickly prune out all non-matching record pairs. However, depending on distributions cluster sizes, existing techniques can be either (a) too aggressive, such that they help scale but adversely affect ER effectiveness, or (b) permissive, potentially harming efficiency. In this paper, we propose new methodology progressive blocking (pBlocking) enable both efficient and effective ER, works seamlessly across different size distributions. pBlocking based insight effectiveness–efficiency trade-off revealed only when output starts available. Hence, leverages partial in feedback loop refine result data-driven fashion. Specifically, bootstrap with traditional methods progressively building scoring blocks until get desired trade-off, leveraging limited amount results as guidance at every round. We formally prove converges efficiently ( $$O(n \log ^2 n)$$ time complexity, where n total number records). Our experiments show incorporating effectiveness by 5 $$\times $$ 60%, respectively, improving overall F-score entire process up 60%.